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Abstract: This paper reports on the design and evaluation of a UK University’s global eLearning MBA 
programme. The aims of the research were to investigate the learning experiences of the students on the course 
and to evaluate the effectiveness of the support system so as to improve the programme. The primary research 
method was a longitudinal semi-structured questionnaire survey, and data were collected from students taking 
the course during the years 2008-2010. Three rounds of survey were conducted, resulting in 149 valid 
responses. The first round showed a fairly high level of student satisfaction with the programme, but also 
indicated areas that needed further improvement. The impacts of subsequent changes in the programme and the 
learning support system were investigated in the second and third rounds of the survey. Feedback from these 
has helped develop additional changes in the learning content and delivery approach of the programme. Overall, 
the findings helped improve the course’s delivery approach, enriched the course’s content, enhanced its quality, 
and improved the satisfaction level of the students. It is hoped that these findings can provide useful insights to 
course managers and eLearning developers of other courses offered in a global context. 
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1. Introduction 

eLearning is still developing rapidly, supported by increasing sophistication of information technology 
and by better understanding of how to make content and delivery of e-courses more effective. 
Moreover, various types of eLearning support system have been increasingly introduced to higher 
education institutions in an effort to meet the student-centred learning paradigms recommended by 
UNESCO (UNESCO, 1998). 

In this context, the creation and implementation of effective quality assurance for such learning 
process has been identified as one of the most challenging tasks. Jara and Mellar (2010) and 
Martinez-Arguelles et al. (2010) point out that the collection of student feedback should be a central 
part of strategies to monitor the quality and standards of teaching and learning in higher education 
institutions for both conventional learning and eLearning. But Jara & Mellar also note that while 
research into eLearning abounds, studies that focus on the effectiveness of the provision of eLearning 
are limited, and that this is a gap to be filled. The research reported below relates to this gap. 

The research focuses on the evaluation of a global work-based eLearning programme - the Executive 
MBA of a UK University. This research has been conducted and is reported here by three members of 
the MBA team responsible for the programme - two Senior Lecturers and the ELearning Development 
Manager. The research focuses specifically on the quality and effectiveness of the developments 
made in the course’s delivery approach and its learning content. The aims were to investigate the 
learning experiences and perceptions of the students, evaluate the effectiveness of the eLearning 
support system, and identify the gaps between the students’ expectations and their actual 
experiences in taking the programme, with a view to improving the programme. Data were collected 
by longitudinal semi-structured questionnaire survey of students taking the course over the past three 
years. 

This paper begins with a review of the literature related to eLearning, evaluation, and quality 
assurance. Then an overview of the MBA eLearning programme is provided. This is followed by a 
discussion of the research methodology, the findings, and the consequent improvements made to the 
course. Research limitations are pointed out, and reflections made on the research. Finally, a 
summary and conclusions are given, followed by research implications. 

2. Literature review 

The rapid growth of online academic course provision worldwide has changed the learning 
environment for both students and teachers (Landry et al., 2008; Lapointe & Reisetter, 2008; Williams 
& Williams, 2010). ELearning has taken many forms, such as fully online, mixed mode or hybrid, 
blended learning and web-assisted (Buzzetto-More 2008). It is claimed that eLearning overcomes 
many drawbacks that are inherent to traditional classroom teaching, especially its lack of flexibility in 
the use of resources (for example, Lam & Bordia 2008; Williams & Williams, 2010). Goold, et al. 
ISSN 1479-4403 1 ©Academic Publishing International Ltd 

Reference this paper as: Bentley, Y, Selassie, H and Shegunshi, A. “Design and Evaluation of Student-Focused 
eLearning” The Electronic Journal of e-Learning Volume 10 Issue 1 2012, (ppOl-12), available online at 
www.eiel.org 




Electronic Journal of e-Learning Volume 10 Issue 1 2012 


(2007) indicate that this type of online learning environment enables a greater number of students of 
diverse educational and cultural backgrounds, as well as of modes of study, to come together within 
the one virtual classroom. However, they warn that many of the clues that help enable staff and 
students to be culturally sensitive in physical classrooms are missing in the online world. For this and 
other reasons, it has been suggested that students need better preparation for learning in an online 
environment than in a traditional classroom. 

According to Rajasingham (2009) and Guri-Rosenblit (2009), eLearning has evolved from distance 
education and is still struggling to gain full recognition and accreditation within mainstream education 
as an approach for high quality provision. While developments in eLearning have been exciting and 
beneficial, finding ways of enhancing the quality of provision and effectiveness have posed a serious 
challenge. In response to this concern of legitimacy, value and quality of online programmes, Davies 
et al. (2011) develop a model that provides a comprehensive conceptual framework which identifies 
the factors that enhance the quality of fully-online degree programmes. Pillay & Kimber (2011) argue 
that globalisation, transnational provision of higher education, and the ‘use of market mechanisms’ 
have increased the complexity in issues of accountability, authority, and responsibility in quality 
assurance. 

Zygouris-Coe, et al. (2009) note that instituting a well-structured quality assurance process can be 
expensive and time consuming, but that it can be worth the effort. For example, the study undertaken 
by Kidney, et al. (2007) supports this. They state that the merit, quality and success of the eLearning 
programme they investigated were mainly due to the proper application of the quality assurance 
strategies. Rajasingham (2009) notes that new educational paradigms and models that challenge 
conventional assumptions and indicators of quality assurance are becoming possible with the help of 
the increasing sophistication in information technology. 

Deepwell (2007) makes a useful distinction between quality assurance and evaluation, and views 
evaluation as an instrument of quality enhancement rather than quality assurance. Wang (2006) 
identified learning effectiveness, access, student satisfaction, faculty satisfaction, and cost- 
effectiveness as the five ‘pillars of quality’ of online programmes. However, Martinez-Arguelles, et al. 
(2010) identify the key quality dimensions - from the student’s point of view - to be the learning 
process, administrative processes, teaching materials and resources. 

We now turn to evaluation more specifically. Evaluation is defined by Saunders (2003) as the 
“purposeful gathering, analysis and discussion of evidence from relevant sources about the quality, 
effectiveness, and impact of provision, development or policy.” (Saunders, 2003: 39). While the 
measurement of student feedback is recognised as an important component of quality assurance, 
there have been mixed reports as to its effectiveness. For example, according to Gurau and Drillon 
(2009), analysing users' perceptions regarding an eLearning system can provide valuable data to 
evaluate and improve its functioning and performance. On the other hand, Jara and Mellar (2010) 
report from their research findings that student feedback was not always fully adequate to support 
quality enhancement. So a researcher is cautioned that they will need to make judgements in this 
area, and maybe conduct further research to validate initial findings. 

Finally, Lapointe & Reisetter (2008) suggest that the new reality of online learning demands a 
reassessment of our understanding of what makes for the most productive student engagement. The 
findings reported below are intended to help move towards an answer to this question. 

3. Overview of the MBA programme 

The Executive MBA programme discussed in this paper is primarily designed for those with a suitable 
management background who wish to become more effective and enterprising managers in the 
knowledge-based global economy. The MBA programme has been delivered for over seven years to 
work-based eLearning students in the UK and at a number of overseas countries including: Oman, 
India, Germany, Poland, South Africa and Switzerland. It is delivered mainly online via the UK 
University’s eLearning learning support system, but also provides periods of face-to-face teaching. 
Therefore, it is also called a ‘blended-learning’ programme. The face-to-face teaching can be taken at 
the overseas counterpart university, or in the UK, or at both. In the cases of the overseas courses, 
such teaching is delivered by both the ‘flying faculty’ (lecturers from the Business School of the UK 
university) and by lecturers of the counterpart local universities in partnership with the UK University’s 
course leaders. Currently there are about 400 students in total taking the course, of which about 40% 
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are based in Oman, 25% in the UK (full-time and part-time students combined), 15% in Poland, 10% 
in South Africa, and remaining 10% in India, Germany and Switzerland combined. 

Students on the course are required to take and pass ten course modules to complete the programme 
- eight taught modules and two applied management projects. It normally takes two years (4 
semesters) for work-based students to complete the course. The revised course structure of this 
programme, based in part on the findings from the first and second surveys of this research, consists 
of five modules in the first year and five in the second, as follows: 

Year 1: Leading and Managing People; Accounting for Leaders; Marketing Products and Services in a 
Dynamic Environment; Mobilising Creativity and Innovation; and a Leadership project. 

Year 2: Strategy; Operations and Project Management; Entrepreneurship and Small Business 
Management; Thriving in a Competitive Global Context, and an Integrated Management project. 

Figure 1 shows the main building blocks of the blended-learning system designed for the programme, 
and demonstrates the range of learning modes that are available to the students. 


Pod cast/ video: experts' 
lectures, discussions, 
interviews, news, etc. 


Online interactive 
learning units 


Work based learning 
units, projects, 
assignments 



Online brainstorming & 
discussions (text and voice 
boards), WIMBA Live classroom, 
WIKI, BLOG etc 


Examinations, 
group activities, 
Individual tasks, etc 


Specifically designed 
PDP for individual 
professionals 


Figure 1: The blended learning structure of the executive MBA programme (Priestman, 2010: 14) 

At the start of each semester the ‘flying faculty’, with the support of the local institution, conduct 
induction sessions for the students of each cohort at the overseas centres, and this is followed by two 
full days of face-to-face teaching. The main purpose of this approach is to give the students an 
overview of each module run in the semester, and a feeling for the level of teaching, and the degree 
of response expected of them, throughout the course. 

The UK lecturers are in charge of preparing and uploading all the online course content, including 
PowerPoint slides, relevant articles and papers, links to useful websites, audio and video clips, 
podcasting clips, assignment instructions, marking and feedback, etc. although tutors from the 
overseas partner institutions can also contribute to online material. The latter institutions, with their 
supporting lecturers and teaching facilities, provide face-to-face teaching once a month throughout 
the programme. These institutions also provide the local basic services to meet the requirements of 
the students. In addition, the UK lecturers deliver regular WIMBA (a voice over the Internet system) 
sessions for each module. WIMBA is a plug-in WebCT Vista which has ‘live classroom’ and ‘voice 
tools’ components by which the tutor and the students in different locations can hold a virtual class 
simultaneously, see each other via the webcam, and talk to each other as in a traditional classroom. 
WIMBA provides a forum, among other things, for UK lecturers to get a good grasp of the progress 
that the students have made via this blended learning approach, answer their questions related to the 
subject, and provide additional support for the coursework. 
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Assignments for all modules are delivered to the students online by the UK lecturers, and the students 
are required to submit all their assignments to the UK lecturers via the Turnitin system integrated with 
the eLearning support system BREO. The Turnitin system is an online plagiarism checker for 
students’ course assignments, and is an effective means to check the extent to which assignments 
submitted for the course are written by the students themselves. Examinations are taken and 
invigilated at the local institutions, but the scripts are posted to the UK. All assignments and 
examination scripts are marked by the UK lecturers. 

4. Methodology: Semi-structured questionnaire survey 

The primary research approach adapted for this study is a longitudinal survey (Dillon et al., 1987), 
using a semi-structured questionnaire to collect data from respondents over time. This involved three 
rounds of surveys conducted across selected course sites at different points in time, thus enabling 
examination of the changes that occurred in the attitudes and satisfaction levels of the students. The 
first survey round covered four sites - Oman, India, UK, and Poland, and was carried out from April to 
July 2008. The second round covered two sites - Oman and India, and took place from January to 
March 2009. The third covered Oman, UK and Germany and was undertaken from June to December 
2010. The survey forms were sometimes delivered online, and sometimes handed out in class, 
depending on what was most convenient. The split of survey respondents by country and by period is 
given in Table 1. There were 149 valid responses in total, with two thirds of these from the Oman site, 
which, as mentioned earlier, has had the largest student population on the programme. 


Table 1: Summary of survey respondents by country (N=149) 



1st Survey (2008) 

2nd Survey (2009) 

3rd Survey (2010) 


% of 


No. of 

% of 

No. of 

% of 

No. of 

% of 

Total 

responses 

Country 

responses 

responses 

responses 

responses 

responses 

responses 

responses 

by country 

Oman 

30 

48% 

30 

81% 

40 

82% 

100 

67% 

India 

18 

29% 

7 

19% 

- 

- 

25 

17% 

UK 

6 

10% 

- 

- 

4 

8% 

10 

7% 

Germany 

- 

- 

- 

- 

5 

10% 

5 

3% 

Poland 

9 

14% 

- 

- 

- 

- 

9 

6% 

Total 

63 

100% 

37 

100% 

49 

100%, 

149 

100% 


A number of pragmatic considerations led to the changes in choice of survey sites over the course of 
the surveys. These included dates and times when the courses were offered; accessibility of the 
students; and, on occasion, the availability of a researcher being available to deliver and collect 
survey forms. This impacted, to some extent, the comparability of data between sites and over time 
periods. However it was felt on balance that there was enough continuity between sites, and student 
cohorts, for useful comparison to be drawn. 

5. Data analysis and discussion of key issues 

In higher education, evaluation can cover many aspects, such as programme design, adequacy of 
resources, performance of students, and of lecturers, effectiveness of policies, etc. For evaluating this 
blended-learning MBA programme, the evaluation process was designed to measure and improve the 
quality and effectiveness of technology-enhanced teaching, and the learning experience of the course. 
The investigation focussed specifically on the areas of course management, learning and teaching, 
online learning content, assessment, the learning support systems in place, and students’ overall 
experience of taking this course. These areas were seen as key for measuring the quality of the 
programme as provided to the students. In particular, as the focus was on improvement in course 
content and provision, the outcomes of such changes themselves needed evaluation. For this reason 
the second and third survey rounds contained a number of different questions from the first. This was 
to take into consideration the findings from the first survey, as well as the changes that had been 
made to the MBA programme as a result of these findings. 

5.1 The first survey 

For the first survey round, as Table 1 shows, 63 completed questionnaires were received from four 
MBA centres - Oman, India, UK and Poland. The students in Poland and UK completed the survey 
online; while those in Oman and India completed it off-line, with the lecturers directly handing the 
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questionnaire to the students and collecting it on the same day. The latter process contributed to a 
higher rate of returns from these centres. An overview of the perceptions of the respondents for this 
first survey round is given in Table 2. 

Table 2: Results from the first survey (% response by category) 


Q 

Survey Items 

Excellent 

Good 

Average 

Poor 

% (E&G) 

% (A&P) 

1 

Registration process 

31 

55 

11 

3 

86 

14 

2 

Teaching at Induction 

20 

60 

16 

4 

80 

20 

3 

Teaching at local institution 

18 

58 

24 

0 

76 

24 

4 

Module Handbooks 

22 

47 

27 

4 

69 

31 

5 

Local Tutor support 

15 

45 

33 

7 

60 

40 

6 

Module information from local institution 

9 

67 

22 

2 

76 

24 

7 

Local support facilities 

9 

51 

26 

14 

60 

40 

8 

Textbook availability 

18 

33 

22 

27 

51 

49 

9 

Usefulness of CD ROMS 

20 

44 

22 

14 

64 

36 

10 

Quality of CD ROM materials 

20 

58 

20 

2 

78 

22 

11 

Reading materials on BREO 

26 

38 

27 

9 

64 

36 

12 

Learning resources/weekly online 

22 

45 

22 

11 

67 

33 

13 

UK Lecturer support 

11 

51 

33 

5 

62 

38 

14 

Slides on BREO 

25 

44 

25 

6 

69 

31 

15 

Audio/video clips 

3 

45 

21 

31 

48 

52 

16 

Relevant website links 

15 

43 

29 

13 

58 

42 

17 

Voice Cafe 

11 

26 

25 

38 

37 

63 

18 

Course/module response 

2 

47 

42 

9 

49 

51 

19 

IT training and support 

11 

35 

42 

12 

46 

54 

20 

Assignment instructions 

16 

53 

27 

4 

69 

31 

21 

Assignment submission procedures 

16 

55 

13 

16 

71 

29 

22 

Assignment feedback 

15 

36 

29 

20 

51 

49 

23 

Referral procedure 

13 

57 

23 

7 

70 

30 

24 

Failure procedures 

18 

39 

17 

26 

57 

43 

25 

Academic offence procedures 

16 

44 

16 

24 

60 

40 

26 

Social networking opportunities 

9 

38 

33 

20 

47 

53 

27 

Overall experience with the tutors 

24 

51 

23 

2 

75 

25 

28 

Overall experience of online support 

8 

49 

36 

7 

57 

43 

29 

Overall experience of undertaking the 
blended learning MBA 

20 

46 

29 

5 

66 

34 


Average: 

16 

47 

25 

12 

f 

63 

w 

37 


The results in this table are self-explanatory. Overall, the survey results showed a fairly high level of 
satisfaction with the MBA programme, with an average of 63% of the respondents perceiving the 
average of the 29 aspects of the course measured as being either ‘excellent’ or ‘good’, versus 25% 
perceiving this average as ‘average’, and 12% as ‘poor’. In terms of the service quality as perceived 
by the students, 22 out of the 29 aspects surveyed showed a combined percentage of ‘excellent’ and 
’good’ (‘% E&G’ in Table 2) to be greater than the combined percentage of ‘average’ and ’poor’ (% 
A&P). Six results had these percentages about the same (questions 8, 15, 18, 19, 22, 26), and one 
(question 17, on Voice Cafe) had this percentage significantly reversed (37% versus 63%). 

As this was a semi-structured questionnaire survey, the respondents were given the opportunity to 
offer comments where appropriate in order to provide more detailed information on the topic areas 
being investigated, and to encourage suggestions for improving the course content, delivery 
approach, and support systems. Examples of positive comments included: 

‘Voice over the Internet, Blackboard, and the student records system are excellent 
facilities for remote students like us. I found this course very balanced and suitable for 
working candidates’. 


www.eiel orq 


5 


ISSN 1479-4403 











































Electronic Journal of e-Learning Volume 10 Issue 1 2012 


7 was impressed with the free wireless Internet facility and the ever-helpful nature of the 
faculty office and administration team’. 

‘It has been really a correct decision for me to enrol onto the MBA programme. The 
wisdom I derived from [taking this] e-MBA is tremendous’. 

‘It was a pleasure and enjoyable experience taking the MBA with this UK University’. 

Not surprisingly, there was also a range of negative comments, and the ratio of positive to negative 
comments was about three to two. To put these comments into context it needs to be borne in mind 
that the students were mostly at middle and senior-level management in full-time employment, and 
were working on the Executive MBA often on day or week release schemes from their employer, and 
at weekends. As a result, student expectations of the course were generally high, and the time 
pressure on the students was often quite onerous. 

The following are examples of some of the criticisms from respondents: 

■ Many respondents asked for more consistent and comprehensive module information and 
guidance, clearer assignment instructions, and for the formats of examinations. Some mentioned 
that they had a desire to have all the essential information about the course issued at the 
induction sessions (rather than having some information fed piecemeal later). 

■ Some felt that BREO had been under-used, e.g. it was not used to provide effective discussion 
forums. 

■ A few respondents expressed a dislike for the different assignment submission systems used on 
different courses within the MBA, as these were often seen as time-consuming and confusing. 

■ Some felt very dissatisfied if they failed to receive their assignment feedback within the specified 
time, but appreciated the cases where more detailed and customised feedback had been given, 
rather than just a brief and general comment, as they wanted to understand what was wrong in 
their work. They also had a desire for quicker responses to requests for information and feedback. 

■ Some felt the need to spend more time with the UK-based tutors, and expressed a wish for an 
additional round of face-to-face interaction in the middle of a semester from the UK course 
leaders. Some would have liked to gain the opportunity for the classroom experience at the UK 
University. 

■ In addition, it was suggested that the communication and collaboration between the education 
partners be improved, so that better and more effective on-line support can be provided. 

Overall, the above comments could be summarised as indicating a need for more support for the 
students’ independent learning process. As can be seen from Table 2, the average of positive 
responses (‘excellent’ plus ‘good’) on support for eLearning itself (questions 4, 6, 10, 12, 20, 21) was 
reasonably high. Nevertheless, the perceptions of individual students showed that there was still room 
in a number of areas for the university to make improvement. Suggestions from the above research, 
together with additional feedback collected from consultation with both the UK and local staff, were 
considered in detail, and subsequently, a significant range of changes to the programme were made. 
Some of these are discussed in the following section. 

5.2 The second and third surveys 

Subsequent to the changes made after the first survey round carried out early in 2008, a second 
survey round was conducted in 2009 covering two overseas sites: Oman and India. This survey had 
28 questions of which 19 were the same as those in the first survey. Thirty-seven valid responses 
were received. The primary purpose of this second round was to measure any improvement (or 
otherwise) resulting from the course changes, and to uncover additional issues. A comparison 
between all the three rounds of surveys is given in Table 3. (For consistency, question numbers 
match those of Table 2.) 

An examination of the results from the first two survey rounds showed that 14 out of the 19 topic 
areas investigated in common across the two surveys had improvements in the second survey in the 
‘positive’ response category (i.e., % of ‘excellent’ plus ‘good’), with nine areas showing a gain of 10% 
or more. Significant improvements included: teaching at induction by the UK ‘flying faculty’; the 
quality of module handbooks; UK tutor support; audio/video clips; ICT training and support; social 
networking opportunities; overall experience of online support; and the overall experience of 
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undertaking the MBA (where the latter response rose from 66% to 79% in the ‘E&G’ category). In 
addition, the percentages in the ‘E&G’ category for both Question 10 (the change from providing 
‘learning materials on CD Rom’ in the first survey to providing ‘weekly online material’ in the second), 
and Question 17 (the change from using ‘Voice Cafe’ for Internet conference in the first survey to the 
adoption of ‘WIMBA’ in the second), rose from 64% to 73% and from 37% to 85%, respectively, after 
the changes were made. 


Table 3: Comparison of equivalent questions from all three surveys (% response by category) 


Q 

Areas covered 

1st Survey results (2008) 

2nd Survey results 

2009) 

3rd Survey results 

2010) 

Excel 

Good 

Av. 

Poor 

E&G 

A&P 

Excel 

Good 

A v. 

Poor 

E&G 

A&P 

Excel 

Good 

Av. 

Poor 

E&G 

A&P 

1 

Registration process/ 
admin support 

31 

55 

ii 

3 

86 

14 

22 

59 

19 

0 

81 

19 

27 

46 

21 

6 

73 

27 

2 

Teaching at 

Induction/Asmin support 

20 

60 

16 

4 

80 

20 

27 

62 

11 

0 

89 

11 

26 

50 

24 

0 

76 

24 

3 

Teaching at local 
institution 

18 

58 

24 

0 

76 

24 

5 

62 

30 

3 

67 

33 

5 

42 

45 

8 

47 

53 

4 

Module Handbooks 

22 

47 

27 

4 

69 

31 

14 

67 

19 

0 

81 

19 

8 

54 

16 

22 

62 

38 

5 

Local Tutorsupport 

15 

45 

33 

7 

60 

40 

30 

27 

36 

7 

57 

43 

22 

56 

11 

11 

78 

' 22 

10 

CD ROM materials; 
weekly online material 

20 

44 

22 

14 

64 

36 

22 

51 

16 

11 

73 

27 

29 

49 

19 

3 

78 

22 

13 

UK tutor support 

11 

51 

33 

5 

62 

38 

49 

32 

16 

3 

81 

19 

22 

48 

30 

0 

70 

' 30 

15 

Audio/video clips 

3 

45 

21 

31 

48 

’52 

25 

61 

13 

1 

' 86 

' 14 

- 

- 

- 

- 

- 

- 

17 

Voice Cafe-WIMBA 

11 

26 

25 

38 

37 

' 63 

53 

32 

14 

1 

' 85 

’ 15 

- 

- 

- 

- 

- 

- 

19 

IT training and support/ 
online guides 

11 

35 

42 

12 

46 

54 

14 

54 

30 

2 

68 

32 

10 

50 

38 

2 

60 

40 

20 

Assignment instructions 

16 

53 

27 

4 

69 

31 

27 

57 

3 

13 

' 84 

’ 16 

- 

- 

- 

- 

- 

- 

21 

Assignment submission 
procedures 

16 

55 

13 

16 

71 

29 

30 

48 

19 

3 

78 

22 

16 

76 

8 

0 

92 

8 

22 

Assignment feedback 

15 

36 

29 

20 

51 

49 

16 

46 

19 

19 

' 62 

38 

- 

- 

- 

- 

- 

- 

23 

Referral procedure 

13 

57 

23 

7 

70 

30 

14 

45 

31 

10 

' 59 

41 

- 

- 

- 

- 

- 

- 

24 

Failure procedures 

18 

39 

17 

26 

57 

43 

19 

26 

40 

15 

' 45 

’ 55 

- 

- 

• 

- 

- 

- 

25 

Academic offence 
procedures 

16 

44 

16 

24 

60 

40 

17 

44 

30 

9 

'- 

61 

f - 

39 

- 

. 

. 

. 

. 

. 

26 

Social networking 
opportunities 

9 

38 

33 

20 

47 

53 

78 

5 

16 

0 

83 

16 

_ 

_ 

. 

. 

. 

. 

28 

Overall expenence of 
online support 

8 

49 

36 

7 

57 

43 

30 

49 

5 

16 

79 

22 

22 

56 

16 

6 

78 

22 

29 

Overall experience of 
undertaking the MBA 

20 

46 

29 

5 

66 

34 

30 

49 

16 

5 

79 

21 

57 

24 

18 

0 

r 

81 

18 


Average of responses 
shown: 

16 

46 

25 

13 

63 

38 

27 

46 

20 

6 

74 

26 

22 

50 

22 

5 

72 

28 


However, it was surprising to see that in spite of the effort made for improvement after the first survey 
some areas did worse, including teaching by local institutions, and the referral procedure and the 
failure procedure. This indicated that further effort was needed to improve the experience of the 
students, though in part it might be that later students were more demanding, especially for services 
related to certain university procedures. Overall, the average percentage of respondents who rated 
the course as ‘excellent’ and ‘good’ on the topics included in this comparison improved from 63% in 
the first survey to 74% in the second. This was seen as a very encouraging finding, and justified the 
extensive work by all the parties involved to improve the design and provision of the course. 

Note that other questions in the second survey were different from those in the first as many changes 
had been made in various aspects of the course after the first survey, and questions on these 
changes were incorporated in second survey to examine the impacts on the students’ learning 
experience. For example, more informative and meaningful module handbooks and ICT training 
guides were introduced, and questions related to these changes were added. 

Some of the more significant changes that had been made to the programme following the first survey 
are discussed below, set against the findings from the two surveys to indicate how the gains 
mentioned above in student satisfaction were achieved. 

The introduction of WIMBA. WIMBA was adopted since August 2008 to replace the Voice cafe system 
(voice over the Internet). WIMBA is an audio component to Blackboard courses enabling users to 
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record voice announcements and audio blogs, send email messages with embedded audio, and 
conduct live voice chat sessions. One student commenting on this after a WIMBA session said that “it 
is the best teaching session after the induction”. Regarding the WIMBA live classroom, the survey 
showed that the respondents who responded in the ‘excellent’ and ‘good’ categories were over 50%, 
and 30%, respectively. The students also liked the idea of archiving the WIMBA sessions as an 
additional eLearning resource, as it allows students (such as those who have missed a class) to see 
the archived files. BREO tracking statistics showed that for the seven WIMBA archives which were 
made available for one cohort of students during March 2009, there were 107 hits, which indicated 
that the students appreciated the effort the university had made to provide this facility. Most 
respondents considered WIMBA as the better eLearning supporting system, as the second survey 
showed that the students’ perception of WIMBA was significantly more positive than that for Voice 
Cafe (85% versus 37%). 

Improved course delivery approach: A significant change made in the blended learning support 
system for the MBA programme was the restructuring of the format and timing of the online learning 
content. This was changed so that all content was presented online in a standard format, and made 
available weekly consistent with the learning schedule. This represented a significant change 
compared to the previous system where the key course material was provided on a CD for each 
module. In comparing the two elements (provision of CD ROMs versus material online weekly), the 
percentage of students viewing course provision as being ‘excellent’ plus ‘good’ increased from 64% 
to 73%. The respondents also liked the audio and video clips prepared by the UK tutors on specific 
topics, and 80% wished to have more such clips for each of the modules. 

Improved course and module information: In the first survey, students expressed their wish that each 
module handbook provide all the information about the module, in particular, including details about 
the exams and assignments, and examples of good assignments. The format of all the module 
handbooks were therefore expanded and standardized, and as a result the students reported that 
they were clearer about requirements for assignments in each module that they took. In terms of 
response to the questionnaire, the positive response level for this area improved from 69% to 81%. 

Provision of online training materials: Subsequent to the first survey, a range of training materials, 
including an ICT guide and audio/video clips for using the university’s digital library were developed 
and made available online to the students. As a result, in the second survey the students rated the 
MBA ICT guide, the WIMBA live classroom guidelines, the clips on digital library, and the Harvard 
referencing guidelines, as the most useful tools, followed by the guide for using Net-mail (the e- 
mailing system for students). Overall, the survey evaluation of ‘excellent’ plus ‘good’ for the training 
and supporting materials available rose from 46% in the first survey to 68% in the second. 

Improved usability of the eLearning tool: As mentioned above, in the first survey one respondent had 
commented that BREO was “the most under-used tool”. However, after the changes were introduced, 
statistical tracking showed very high hit rate for the weekly online material for all the modules. For 
example, a total of 182 students from four randomly selected modules in Oman (semester 2 of the 
academic year 2008/9) showed a total of over 8047 hits within one week. These included the weekly 
online course content (6566 hits), WIMBA live classroom (657 hits), and ICT training guide (824 hits). 
This meant an average of 22 hits per student per week, more than doubling the number of hits (9 hits 
per student per week) before the change was made. 

Additional online contact time: For each module, the UK lecturers on the programmes were assigned 
additional online contact hours to provide better support to the students, including the WIMBA 
sessions as discussed above. Comparison of the surveys indicated that student positive satisfaction 
of the online tutor support improved from 62% in the first survey to 81% in the second. 

Assessment feedback: In terms of providing students with feedback on their assignments, the first 
survey showed 51% of the respondents rated this either ‘excellent’ or ‘good’. Subsequently, a great 
deal of effort was made to improve this area. There was improvement in this rating (to 62%), but this 
was still a fairly low score for a topic on which so much effort had been made. Some students felt that 
for some assignments the feedback was too general and not enough to help them improve future 
assignments. However, many students also acknowledged that some lecturers gave very good 
feedback. Clearly, given the constraints on lecturers’ time in all such courses, fully satisfying student 
demands in this area will always be a challenge. 
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Other topics: Other conclusions from the surveys included the fact that respondents were not keen on 
writing wiki-text and blogs on Blackboard (and neither were the lecturers!). In addition, students really 
disliked receiving the information on plagiarism and the use of Turnitin for submitting assignments 
(over 80% said ‘no’, only 9% said ‘yes’). By contrast, they much liked the examples of good and bad 
assignments for each module (80% said ‘very useful’ or ‘useful’); and while responses on the topics of 
referral, failure, and academic offence procedures, though positive, were not especially so. 

In terms of the students’ overall experience of this blended-learning MBA, while the first survey 
showed that 66% of students graded this as ‘excellent’ or ‘good’, in the second this increased to 79% 
as a result of changes to the programme and its delivery. This was very encouraging, though - 
interestingly - not quite as good as the course providers had hoped, which indicated that there was 
more work to do. 

5.3 The third survey 

In 2010, a third round of surveys was carried out, covering the Oman, Germany and UK sites. Forty- 
nine valid responses were obtained. A number of questions in the third round were different again 
from the second, but there were 11 questions which covered the same areas as in the first and 
second surveys (see Table 3).The MBA programme had been further changed following the second 
survey, but not radically in most areas, so for many of the questions that were the same it is not 
surprising that the results were fairly similar. For example, for the key question that asked about the 
students’ overall experience of the MBA, there was a marginal improvement (from 79% to 81%), 
though this is unlikely to be significant. Some other findings were disappointing. For example, 
evaluation of the teaching at the local partner universities was again rated lower than previously 
(down to just under half of responses being in the ‘E&G’ category), but it is worth noting that over 80% 
of the responses of the second and the third rounds of the survey was from the same overseas site 
(see Table 1) which clearly show that the UK university needs to take some action to improve the 
situation. This fall in rating was partly counterbalanced, however, by a large jump in the rating of 
quality of local tutor support, reflecting the very significant effort that had been put into improving this 
area. 

The third survey round comprised quite a number of individual surveys. For example, at the Oman 
site there was a mid-course on-line survey, and also a mid-course ‘off-line’ (hand-out and return) 
survey. In addition, students were asked to fill in an online exit survey at the completion of their 
course. And, as mentioned earlier, the questionnaires were ‘semi-structured’ in the sense that 
respondents could add comments to clarify or expand their questionnaire answers. Indeed, 
respondents were specifically encouraged to make comments (whether positive or negative) where 
they felt this could improve future course provision. All replies were useful. Most were fairly 
straightforward; and of course in some cases asked for additional resources (more tutor time, for 
example, or faster response on assignment marking) that were unlikely to be met without a 
fundamental change in level of staff provision. Some of the responses were particularly interesting. 
For example, Figure 1 indicates the response at one overseas site to the question: ‘How much do you 
feel part of the UK University?’ The positive reply was seen as very encouraging in terms of students’ 
alignment with the course. 



5-high 4 3 2 1 - low 


Figure 1: Number of responses by category to the question: ‘How much do you feel part of the 
university?’ 
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To a question about the credibility of prior publicity for the course, the response again was fairly 
positive, as indicated in Figure 2. 



Figure 2: Number of responses by category to the question: ‘Is the course as you were led to believe 
by the publicity you saw prior to enrolment?’ 

To the general question: ‘Do you think there is any missing learning element you think should have 
been there as a part of MBA?’ replies included: 

“A little more on entrepreneurship and a little less on leadership issues”. 

“Business School should be twice a month ... with a shorter time, instead of long gruelling 
hours with marathon lessons once a month. I found it hard to concentrate in the 
afternoon classes. ” 

“I would have liked to have more ... focus on HR capabilities as I think this is missing, 
even though the Leadership module tries to cover some of the aspects.” 

“There should be more case studies to be reviewed in class. The Entrepreneurship 
module was taught well through case studies.” 

And finally, one of issues raised related to the perennial question of assigning the proportion of group 
work to Individual work. One respondent put his concern eloquently: “The grave challenge I faced is 
the group assignments. Success of team work depends on choosing right members for the team. 
Everybody just formed teams in the first business school weekend where we didn’t have any idea 
about each other. The collectivist mentality of the local culture played a negative role in working with 
assignments. Free-riders carried on with ease at the expense of my own time and sweat...” 

Nevertheless, overall, the third round of surveys (and the two preceding ones) gave a generally 
positive picture of how the students felt about the course, with some highly complementary remarks 
being made when students were asked to summarise their general satisfaction with the course. 
Additional analyses of the survey findings are on-going which will help further improvements to be 
made. 

6. Limitations and reflections on the research 

Like all research, this research has a number of limitations. Perhaps the main one was the change in 
sites being surveyed at each round. As mentioned earlier, pragmatic considerations had led to these 
changes in survey sites, including the timings of when courses were offered, and suitable occasions 
on which student surveys could be carried out without interrupting teaching schedules. For this reason, 
the comparability of responses between survey rounds needs to be handled with some caution. But 
set against this, in general the courses were being taught to students of a rather similar level across 
all sites and with similar expectations, such that it is felt that, on balance, useful conclusions could be 
drawn. 

A second limitation was response rates, though these were fairly good for a survey of this kind. It had 
been decided at the outset not to make the surveys mandatory in order to ensure responses only 
where students were motivated to respond. But this opened up the possibility, indeed the likelihood, of 
‘self-selection’, responses coming from more active students and those with especially good or bad 
experiences to report. At Oman the questionnaires were mainly on a ‘hand-out and return’ basis and 
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here the response rate across all three surveys was about 75% for the particular cohort surveyed 
(totalling typically about 40 students each cohort). But Oman had four such cohorts studying in their 
different semester of the two year course, so the response rate across all students would fall to just 
below 20%. There was no reason to think other cohorts would respond to the questions differently, 
but the fact is that they were not surveyed. At the other sites, the student numbers were smaller (India 
typically about 30; UK about 13; Germany 9; and Poland 37) such that the response rates were about 
60% (first survey) and 25% (second survey) for India, 40% for UK, 60% for Germany, and 25% for 
Poland. 

In terms of continuity of the survey, there were indeed changes in the question sets over time. But this 
is not thought to be a major impediment to the overall value of the surveys. This was because the 
questions were largely straightforward, and mostly did not involve a high emotional context, where 
specific phasing of the question might have affected the answers given. In addition, most of the 
changes to the question sets were determined by factors such as being no longer relevant (the 
situation having already changed), or where new questions were designed to measure changes made 
after the previous survey round. 

In summary, common-sense considerations (including directness of the questions, number of 
students surveyed, and non-mandatory nature of the surveys) indicate that the results are robust, but 
it has not been possible to quantify this. The surveys will continue, and as more results become 
available greater effort will be put into ensuring inter-comparability of the survey findings. 

7. Conclusions 

The paper reports on the design and evaluation of a blended-learning MBA programme provided by a 
UK university at a number of partner-institution sites around the world. The evaluation was based on a 
longitudinal study involving a sequence of surveys of students’ perceptions of the course. Findings 
from these surveys, plus other internal review procedures, were used to make changes in both course 
content and course delivery approach. 

The research is contrasted with much of the research into the evaluation of eLearning courses which 
has relied on data gathered just once, and where cross-sectional designs have been applied. By 
contrast, the longitudinal approach employed here has enabled the charting of changes over time, 
thus enriching the process of course design, and monitoring the changing satisfaction of students and 
other stakeholders. 

Overall, the survey findings indicated a high level of satisfaction with the MBA programme, and this 
satisfaction increased following the changes made to the programme. In particular, the findings 
supported the many detailed choices made by both the providing university and the partnering 
institutions in the provision of structure and content of the blended-learning approach adopted. 

The findings also give credence to the view that such an approach is particularly suited to 
postgraduate-level MBA provision for students currently in management employment, because such 
students are relatively mature, already business-aware, conversant with information technology, and 
have access to the eLearning facilities and resources required (Priestman 2010). The findings 
furthermore support the claim of Gurau and Drillon (2009) that student feedback can provide valuable 
data to evaluate and improve the functioning and performance of an eLearning system. 

Rajasingham (2009) concluded that educational paradigms that challenge conventional teaching 
methods are becoming increasingly possible with the help of the increasing sophistication in 
information technology. The authors of the current research recognise that student satisfaction 
reported for this course emanate in no small measure from design and implementation approaches 
made possible by this improvement in information technology. Moreover, by identifying areas for 
course improvement, this research has attempted to implement the concept of ‘evaluation as an 
instrument of quality enhancement, rather than just quality assurance’, as suggested by Deepwell 
(2007). 

Finally, it is hoped that the outcomes of the evaluation of eLearning in this research have improved 
not only the quality and effectiveness of this particular programme, but also will provide valuable 
guideline to help improve the quality and effectiveness of the teaching and learning processes 
(Phipps, 1999) of global eLearning or blended courses offered by other institutions across the world. 
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